WIP: TDNN+LSTM and TDNN scripts by vijayaditya · Pull Request #1212 · kaldi-asr/kaldi

vijayaditya · 2016-11-23T20:40:42Z

This PR has been created to keep people informed about the latest TDNN+LSTM and TDNN experiments.
ami : added xconfig TDNN and TDNN+LSTM scripts for AMI.
Results soon.

swbd : added results for the TDNN recipe which uses the subset-dim option (worse than normal TDNN)

+#System                  tdnn_7h   tdnn_7j
+#WER on train_dev(tg)      13.84     14.15
+#WER on train_dev(fg)      12.84     12.96
+#WER on eval2000(tg)        16.5      16.8
+#WER on eval2000(fg)        14.8      15.1
+#Final train prob     -0.0889771-0.0910883
+#Final valid prob      -0.113102 -0.112464
+#Final train prob (xent)       -1.2533  -1.31768
+#Final valid prob (xent)      -1.36743  -1.41603

vijayaditya · 2016-11-24T05:54:20Z

Good results even in AMI.

#old tdnn baseline
%WER 40.6 | 14891 94509 | 63.0 19.8 17.2 3.7 40.6 65.1 | 0.594 | exp/sdm1/chain_cleaned/tdnn_sp_bi_ihmali/decode_dev/ascore_9/dev_hires_o4.ctm.filt.sys
%WER 44.8 | 14251 89975 | 58.7 21.1 20.2 3.5 44.8 64.3 | 0.558 | exp/sdm1/chain_cleaned/tdnn_sp_bi_ihmali/decode_eval/ascore_9/eval_hires_o4.ctm.filt.sys

# new tdnn baseline (i.e., w/o pretraining and using swbd/tdnn_7h architecture)
%WER 39.9 | 14249 94516 | 63.8 19.6 16.6 3.6 39.9 68.1 | 0.589 | exp/sdm1/chain_cleaned/tdnn1b_sp_bi_ihmali/decode_dev/ascore_9/dev_hires_o4.ctm.filt.sys
%WER 43.9 | 13109 89968 | 59.5 21.2 19.3 3.4 43.9 69.8 | 0.557 | exp/sdm1/chain_cleaned/tdnn1b_sp_bi_ihmali/decode_eval/ascore_9/eval_hires_o4.ctm.filt.sys

# tdnn+lstm (based on swbd/tdnn_lstm_1b) 
%WER 38.9 | 14798 94506 | 65.0 19.5 15.5 3.9 38.9 64.9 | 0.636 | exp/sdm1/chain_cleaned/tdnn_lstm1a_sp_bi_ihmali_ld5/decode_dev/ascore_9/dev_hires_o4.ctm.filt.sys
%WER 42.2 | 13507 89988 | 61.3 21.3 17.4 3.5 42.2 67.7 | 0.602 | exp/sdm1/chain_cleaned/tdnn_lstm1a_sp_bi_ihmali_ld5/decode_eval/ascore_9/eval_hires_o4.ctm.filt.sys

danpovey · 2016-11-24T05:58:15Z

Wow, great!

…

On Thu, Nov 24, 2016 at 12:54 AM, Vijayaditya Peddinti < ***@***.***> wrote: Good results even in AMI. #old tdnn baseline %WER 40.6 | 14891 94509 | 63.0 19.8 17.2 3.7 40.6 65.1 | 0.594 | exp/sdm1/chain_cleaned/tdnn_sp_bi_ihmali/decode_dev/ascore_9/dev_hires_o4.ctm.filt.sys %WER 44.8 | 14251 89975 | 58.7 21.1 20.2 3.5 44.8 64.3 | 0.558 | exp/sdm1/chain_cleaned/tdnn_sp_bi_ihmali/decode_eval/ascore_9/eval_hires_o4.ctm.filt.sys # new tdnn baseline (i.e., w/o pretraining and using swbd/tdnn_7h architecture) %WER 39.9 | 14249 94516 | 63.8 19.6 16.6 3.6 39.9 68.1 | 0.589 | exp/sdm1/chain_cleaned/tdnn1b_sp_bi_ihmali/decode_dev/ascore_9/dev_hires_o4.ctm.filt.sys %WER 43.9 | 13109 89968 | 59.5 21.2 19.3 3.4 43.9 69.8 | 0.557 | exp/sdm1/chain_cleaned/tdnn1b_sp_bi_ihmali/decode_eval/ascore_9/eval_hires_o4.ctm.filt.sys # tdnn+lstm (based on swbd/tdnn_lstm_1b) %WER 38.9 | 14798 94506 | 65.0 19.5 15.5 3.9 38.9 64.9 | 0.636 | exp/sdm1/chain_cleaned/tdnn_lstm1a_sp_bi_ihmali_ld5/decode_dev/ascore_9/dev_hires_o4.ctm.filt.sys %WER 42.2 | 13507 89988 | 61.3 21.3 17.4 3.5 42.2 67.7 | 0.602 | exp/sdm1/chain_cleaned/tdnn_lstm1a_sp_bi_ihmali_ld5/decode_eval/ascore_9/eval_hires_o4.ctm.filt.sys — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1212 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVu6C0c5GbxRpmyqAV0Kp5nboQJk3Eks5rBSaRgaJpZM4K7Bwi> .

vijayaditya · 2016-11-25T18:15:54Z

egs/ami/s5b/local/chain/tuning/run_tdnn_lstm_1c.sh

+#!/bin/bash
+
+# same as 1a, but with more TDNN layers between each LSTM
+#System            1a        1c


adding more TDNNs in between lstms improves results while adding TDNNs after LSTMs made it worse.

danpovey · 2016-11-25T19:25:09Z

Interesting. BTW, merge this whenever you see fit.

…

On Fri, Nov 25, 2016 at 1:15 PM, Vijayaditya Peddinti < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In egs/ami/s5b/local/chain/tuning/run_tdnn_lstm_1c.sh <#1212 (review)>: > @@ -0,0 +1,292 @@ +#!/bin/bash + +# same as 1a, but with more TDNN layers between each LSTM +#System 1a 1c adding more TDNNs in between lstms improves results while adding TDNNs after LSTMs made it worse. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1212 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVu1SPlDCJS4bZu9En0O3g5E8nwPmxks5rByXbgaJpZM4K7Bwi> .

vijayaditya · 2016-11-28T16:30:57Z

Results on IHM. Gains are consistent.

TDNN+LSTM
%WER 21.1 | 13098 94474 | 81.7 10.0 8.3 2.8 21.1 54.0 | -0.033 | exp/ihm/chain_cleaned/tdnn_lstm1g_sp_bi_ld5/decode_dev/ascore_11/dev_hires.ctm.filt.sys
%WER 20.9 | 12643 89977 | 81.6 11.3 7.1 2.5 20.9 52.0 | 0.057 | exp/ihm/chain_cleaned/tdnn_lstm1g_sp_bi_ld5/decode_eval/ascore_11/eval_hires.ctm.filt.sys

TDNN
%WER 22.4 | 13098 94487 | 80.6 10.9 8.5 3.0 22.4 55.4 | 0.084 | exp/ihm/chain_cleaned/tdnn_sp_bi/decode_dev/ascore_10/dev_hires.ctm.filt.sys
%WER 22.6 | 12643 89973 | 80.2 12.7 7.1 2.7 22.6 53.1 | 0.158 | exp/ihm/chain_cleaned/tdnn_sp_bi/decode_eval/ascore_10/eval_hires.ctm.filt.sys

vijayaditya · 2016-11-28T19:54:13Z

# 1i is same as 1g but with TDNN output dim 1024 instead of 512
# (num-params 1g:21309812 1i: 43447156)

System               1g          1i
WER on dev        38.3      37.6
WER on eval        41.6      40.9
Final train prob      -0.138017 -0.114135
Final valid prob      -0.238659 -0.245208
Final train prob (xent)      -1.66834  -1.47648
Final valid prob (xent)      -2.17419  -2.16365

danpovey · 2016-11-28T20:01:04Z

nice!

vijayaditya · 2016-11-28T20:03:48Z

Overall tuning of tdnn+lstm recipe gave a 2% abs. reduction in WER.
Compared to 1a, 1i has

3 TDNN layers between successive LSTM layers, instead of 2
It uses a smaller mini-batch size (128 vs 64) which was found to be helpful in TDNN+LSTMs but not in TDNNs
It uses a lot many more parameters (TDNN output dim 512 vs 1024)

System               1a       1i
WER on dev        38.9      37.6
WER on eval        42.2      40.9
Final train prob      -0.142585 -0.114135
Final valid prob      -0.251197 -0.245208
Final train prob (xent)      -1.73176  -1.47648
Final valid prob (xent)      -2.26965  -2.16365

Edited : Previous comparison was with 1b, but it should have been with 1a.

naxingyu · 2016-11-29T07:01:21Z

This result looks good. Will it be helpful for fisher_swbd? I can have it running when I have computing resources... Xingyu 在 2016/11/29 4:01, Daniel Povey 写道:

…

nice! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1212 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADKpxHwi4vUwHz_WQeSeMN-ELl5K3Gzoks5rCzMBgaJpZM4K7Bwi>.

vijayaditya · 2016-11-29T07:08:52Z

We expect it to help there too. You can look at swbd/tdnn_lstm_1b.sh recipe or wait for 1d which is being tested now and expected to be better (based on AMI results).

…

--Vijay

On Tue, Nov 29, 2016 at 2:01 AM, Xingyu Na ***@***.***> wrote: This result looks good. Will it be helpful for fisher_swbd? I can have it running when I have computing resources... Xingyu 在 2016/11/29 4:01, Daniel Povey 写道: > > nice! > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#1212 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe- auth/ADKpxHwi4vUwHz_WQeSeMN-ELl5K3Gzoks5rCzMBgaJpZM4K7Bwi>. > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1212 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADtwoO76yPCzaFncEVSX7R63g27YZqDYks5rC83DgaJpZM4K7Bwi> .

danpovey · 2016-11-29T21:39:00Z

You might want to wait a few days until we check in the 'fast-lstm' code and scripts. Then your experiments would finish a lot faster. On Tue, Nov 29, 2016 at 2:08 AM, Vijayaditya Peddinti < notifications@github.com> wrote:

…

We expect it to help there too. You can look at swbd/tdnn_lstm_1b.sh recipe or wait for 1d which is being tested now and expected to be better (based on AMI results). --Vijay On Tue, Nov 29, 2016 at 2:01 AM, Xingyu Na ***@***.***> wrote: > This result looks good. Will it be helpful for fisher_swbd? I can have > it running when I have computing resources... > > Xingyu > > > 在 2016/11/29 4:01, Daniel Povey 写道: > > > > nice! > > > > — > > You are receiving this because you are subscribed to this thread. > > Reply to this email directly, view it on GitHub > > <#1212 (comment)>, > > or mute the thread > > <https://github.com/notifications/unsubscribe- > auth/ADKpxHwi4vUwHz_WQeSeMN-ELl5K3Gzoks5rCzMBgaJpZM4K7Bwi>. > > > > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#1212 (comment)>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/ ADtwoO76yPCzaFncEVSX7R63g27YZqDYks5rC83DgaJpZM4K7Bwi> > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1212 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVuxZXOOgQQOiuxNaYxWZh2usuaI_-ks5rC8-GgaJpZM4K7Bwi> .

vijayaditya · 2016-11-29T21:57:01Z

@danpovey Is it possible to move Vimal's code or your code as a branch to the main Kaldi repo, so that we can all push our changes to that branch rather than branching from each other's PRs ?

danpovey · 2016-11-29T22:03:14Z

OK, I'll push to 'fast_lstm' branch in main repo; includes Vimal's changes.

…

On Tue, Nov 29, 2016 at 4:57 PM, Vijayaditya Peddinti < ***@***.***> wrote: @danpovey <https://github.com/danpovey> Is it possible to move Vimal's code or your code as a branch to the main Kaldi repo, so that we can all push our changes to that branch rather than branching from each other's PRs ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1212 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVuxNFgq0onH0nc-6nLMnROUHZJ2bBks5rDJ-vgaJpZM4K7Bwi> .

swbd : added results for the TDNN recipe which uses the subset-dim option

vijayaditya commented Nov 25, 2016

View reviewed changes

vijayaditya force-pushed the tdnn_lstm branch from d4b03e0 to 44a848b Compare November 30, 2016 08:16

ami : added TDNN+LSTM scripts for AMI.

426e0d6

swbd : added results for the TDNN recipe which uses the subset-dim option

vijayaditya force-pushed the tdnn_lstm branch from 44a848b to 426e0d6 Compare November 30, 2016 08:18

vijayaditya merged commit 4435248 into kaldi-asr:master Nov 30, 2016

Conversation

vijayaditya commented Nov 23, 2016

Uh oh!

vijayaditya commented Nov 24, 2016

Uh oh!

danpovey commented Nov 24, 2016 via email

Uh oh!

vijayaditya Nov 25, 2016

Choose a reason for hiding this comment

Uh oh!

danpovey commented Nov 25, 2016 via email

Uh oh!

vijayaditya commented Nov 28, 2016

Uh oh!

vijayaditya commented Nov 28, 2016

Uh oh!

danpovey commented Nov 28, 2016

Uh oh!

vijayaditya commented Nov 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

naxingyu commented Nov 29, 2016 via email

Uh oh!

vijayaditya commented Nov 29, 2016 via email

Uh oh!

danpovey commented Nov 29, 2016 via email

Uh oh!

vijayaditya commented Nov 29, 2016

Uh oh!

danpovey commented Nov 29, 2016 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vijayaditya commented Nov 28, 2016 •

edited

Loading